TRANSFORMER-BASED MODELS

KD-Boost: Boosting Real-Time Semantic Matching in E-commerce with Knowledge Distillation (2023.emnlp-industry)

Challenge:	Existing datasets for code search are monolingual, but their query data are only in English.
Approach:	They construct a multilingual code search dataset in four natural and four programming languages using a neural machine translation model and apply back-translation data filtering to it.
Outcome:	The proposed model pre-trained with all natural and programming language data performs best under almost all settings.

Copied to clipboard

Sanjay Agrawal, Vivek Sembium, Ankith M S

Gender bias amplification during Speed-Quality optimization in Neural Machine Translation (2021.acl-short)

Challenge:	Existing SOTA techniques for semantic matching are mostly based on Siamese networks.
Approach:	They propose a novel knowledge distillation algorithm designed for real-time semantic matching . they train low latency accurate student models by leveraging soft labels from a teacher model .
Outcome:	The proposed algorithm outperforms teacher and SOTA knowledge distillation benchmarks on e-commerce datasets.

Copied to clipboard

Adithya Renduchintala, Denise Diaz, Kenneth Heafield, Xian Li, Mona Diab

Multilinguality

Challenge:	et al., 2002) show that gendered noun translation performance degrades faster than BLEU.
Approach:	They propose to use greedy search, quantization, AANs and shallow decoders to speed up decoding . they find minimal degradation of BLEU, but gendered noun translation degrades faster .
Outcome:	The proposed model degrades gendered noun translation performance faster than other models.

Centrality-aware Product Retrieval and Ranking (2024.emnlp-industry)

Copied to clipboard

Hadeel Saadany, Swapnil Bhosale, Samarth Agrawal, Diptesh Kanojia, Constantin Orasan, Zhe Wu

InterpreT: An Interactive Visualization Tool for Interpreting Transformers (2021.eacl-demos)

Challenge:	Ambiguity and complexity of user queries often lead to mismatch between user’s intent and retrieved product titles or documents.
Approach:	They propose a user-intent centrality optimization approach which optimizes for the user intent in semantic product search.
Outcome:	The proposed approach improves product ranking efficiency for ambiguous queries and lexical terms with alphanumeric characters.

Copied to clipboard

Vasudev Lal, Arden Ma, Estelle Aflalo, Phillip Howard, Ana Simoes, Daniel Korat, Oren Pereg, Gadi Singer, Moshe Wasserblat

Transformer-based Approach for Predicting Chemical Compound Structures (2020.aacl-main)

Challenge:	Using Transformer-based models for NLU/NLP tasks is a growing interest . but there are many open questions regarding the behavior of these models .
Approach:	They present an interactive visualization tool for interpreting Transformer-based models.
Outcome:	The tool can track and visualize token embeddings through each layer of a Transformer, highlight distances between certain token embeds, and identify task-related functions of attention heads using new metrics.

Copied to clipboard

Yutaro Omote, Kyoumoto Matsushita, Tomoya Iwakura, Akihiro Tamura, Takashi Ninomiya

Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times? (2023.tacl-1)

Challenge:	Existing methods to predict chemical compound structures from their names are limited and use handcrafted rules.
Approach:	They propose a Transformer-based model that predicts SMILES strings from chemical compound names instead of handcrafted rules.
Outcome:	The proposed model achieves higher F-measures than the existing model and the existing one.

Copied to clipboard

Byung-Doh Oh, William Schuler

Effective Adversarial Regularization for Neural Machine Translation (P19-1)

Challenge:	Existing studies have shown that larger pre-trained language models with more parameters and lower perplexity are less predictive of human reading times.
Approach:	They propose to use a transformer-based model with more parameters and lower perplexity to investigate why these models are less predictive of human reading times.
Outcome:	The results show that the larger models with more parameters and lower perplexity are less predictive of human reading times and eye-gaze durations collected during naturalistic reading.

Copied to clipboard

Motoki Sato, Jun Suzuki, Shun Kiyono

Robustness in NLP

Challenge:	Existing (small) perturbations that induce a critical prediction error in machine learning models are often referred to as adversarial examples.
Approach:	They propose to use adversarial perturbations to regularize text classification tasks by adding adversarials to a typical NMT model structure.
Outcome:	The proposed method significantly improves performance of NMT models, such as LSTM-based and Transformer-based models.

Comparing Probabilistic, Distributional and Transformer-Based Models on Logical Metonymy Interpretation (2020.aacl-main)

Copied to clipboard

Giulia Rambelli, Emmanuele Chersoni, Alessandro Lenci, Philippe Blache, Chu-Ren Huang

FastSeq: Make Sequence Generation Faster (2021.acl-demo)

Challenge:	Logical metonymies are type clashes between an event-selecting verb and an entity-denoting noun . they are typically interpreted by inferring a hidden event on the basis of contextual cues .
Approach:	They propose to use probabilistic and distributional models to model logical metonymy interpretation . they compare models with the best Transformer-based models and some traditional distributional ones .
Outcome:	The proposed models perform well on a complex scenario, but low performance on some datasets suggests that logical metonymy is still a challenging phenomenon for computational modeling.

Copied to clipboard

Yu Yan, Fei Hu, Jiusheng Chen, Nikhil Bhendawade, Ting Ye, Yeyun Gong, Nan Duan, Desheng Cui, Bingyu Chi, Ruofei Zhang

Enhancing Self-Attention via Knowledge Fusion: Deriving Sentiment Lexical Attention from Semantic-Polarity Scores (2024.starsem-1)

Challenge:	Transformer-based models have made tremendous impact in natural language generation, but inference speed is still a bottleneck due to large model size and intensive computing involved in auto-regressive decoding process.
Approach:	They propose an attention cache optimization, an efficient algorithm for detecting repeated n-grams, and an asynchronous generation pipeline with parallel I/O to accelerate sequence generation without loss of accuracy.
Outcome:	The proposed framework can accelerate the sequence generation by 4x to 9x with a simple one-line code change for a set of widely used and diverse models.

Copied to clipboard

Dongjun Jang, Jinwoong Kim, Hyopil Shin

Commentary Generation from Data Records of Multiplayer Strategy Esports Game (2024.naacl-srw)

Challenge:	Existing methods to inject lexical features into self-attention mechanisms have shown remarkable performance across various downstream tasks in NLP.
Approach:	They propose to inject lexical features into the self-attention mechanism of Transformer-based models by injecting lexicon-based Sentiment Lexical Attention into the attention scores throughout the training process.
Outcome:	The proposed method shows significant performance improvements on the NSMC sentiment classification benchmark and is able to perform in out-of-domain tasks.

Copied to clipboard

Zihan Wang, Naoki Yoshinaga

IrEne-viz: Visualizing Energy Consumption of Transformer Models (2021.emnlp-demo)

Challenge:	Esports play logs are expensive for human experts to provide individual games with play-by-play commentaries.
Approach:	They first build large-scale esports data-to-text datasets that pair structured data and commentaries from a popular eSports game, League of Legends.
Outcome:	The proposed model can generate game commentaries from esports’ data records while examining the impact of the pre-trained language models.

Copied to clipboard

Yash Kumar Lal, Reetu Singh, Harsh Trivedi, Qingqing Cao, Aruna Balasubramanian, Niranjan Balasubramanian

Multiformer: A Head-Configurable Transformer-Based Model for Direct Speech Translation (2022.naacl-srw)

Challenge:	IrEne is an energy prediction system that accurately predicts inference energy consumption of transformer-based NLP models.
Approach:	They present an online platform for visualizing and exploring energy consumption of transformer-based NLP models.
Outcome:	The proposed system predicts energy consumption of transformer-based models and their components.

Copied to clipboard

Gerard Sant, Gerard I. Gállego, Belen Alastruey, Marta Ruiz Costa-jussà

Friendly Topic Assistant for Transformer Based Abstractive Summarization (2020.emnlp-main)

Challenge:	Existing approaches to address speech tasks with a self-attention mechanism are expensive and lead to information loss.
Approach:	They propose a Transformer-based model which uses different attention mechanisms on each head to bias the self-attention towards the extraction of more diverse token interactions.
Outcome:	The proposed model outperforms baseline models by 0.7 BLEU in the speech task.

Copied to clipboard

Zhengjue Wang, Zhibin Duan, Hao Zhang, Chaojie Wang, Long Tian, Bo Chen, Mingyuan Zhou

Fast Whitespace Correction with Encoder-Only Transformers (2023.acl-demo)

Challenge:	Abstractive document summarization is a comprehensive task in natural language processing.
Approach:	They propose a topic assistant that rearranges and learns document semantics . they propose TA that is compatible with Transformer-based models and user-friendly .
Outcome:	The proposed model is compatible with Transformer-based models and user-friendly.

Copied to clipboard

Hannah Bast, Matthias Hertel, Sebastian Walter

The Role of Syntactic Planning in Compositional Image Captioning (2021.eacl-main)

Challenge:	a new tool for whitespace correction is available for text with spurious spaces . the tool is 900 times faster than the previous best tool for text correction .
Approach:	They propose to combine whitespace correction with a character-level encoder-decoder model and a byte-level byte encoder only model to improve quality.
Outcome:	The proposed tool is over 900 times faster than the previous best tool, with the same high quality.

Copied to clipboard

Emanuele Bugliarello, Desmond Elliott

Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation (2020.findings-emnlp)

Challenge:	Image captioning is a core task in multimodal NLP, where the aim is to automatically describe the content of an image in natural language.
Approach:	They propose to use syntactic tags and tokens to improve caption generalization . they also propose to model the syntakic structure of a caption to improve generalization.
Outcome:	The proposed models improve generalization and performance on standard metrics while requiring syntactic and semantic knowledge of the language.

Copied to clipboard

Alessandro Raganato, Yves Scherrer, Jörg Tiedemann

Claim Extraction and Law Matching for COVID-19-related Legislation (2022.lrec-1)

Challenge:	Recent studies have shown that attention heads learn simple positional patterns .
Approach:	They propose to replace all but one attention head of each encoder layer with simple fixed – non-learnable – attentive patterns that are solely based on position and do not require external knowledge.
Outcome:	The proposed model improves translation quality and improves BLEU scores by up to 3 points in low-resource scenarios.

Copied to clipboard

Niklas Dehio, Malte Ostendorff, Georg Rehm

XDBERT: Distilling Visual Information to BERT from Cross-Modal Systems to Improve Language Understanding (2022.acl-short)

Challenge:	Existing approaches to extract legal claims from news articles and match them with applicable laws are difficult for laypersons to learn since news articles do not refer to underlying laws.
Approach:	They propose an automated approach to extract legal claims from news articles and match the claims with applicable laws.
Outcome:	The proposed model achieves 46.7 F1 for claim extraction and 91.4 F1 law matching, despite conceptual limitations.

Copied to clipboard

Chan-Jan Hsu, Hung-yi Lee, Yu Tsao

A Primer in BERTology: What We Know About How BERT Works (2020.tacl-1)

Challenge:	XDBERT (cross-modal distilled BERT) outperforms pretrained-BERT in general language understanding evaluation (GLUE), situations with adversarial generations (SWAG) benchmarks, and readability benchmarks.
Approach:	They propose to distill visual information from pretrained multimodal transformers to pretrained language encoders to cater to the language-heavy characteristics of NLU.
Outcome:	The proposed framework outperforms pretrained-BERT in general language understanding evaluation (GLUE), situations with adversarial generations (SWAG), and readability benchmarks.

Copied to clipboard

Anna Rogers, Olga Kovaleva, Anna Rumshisky

Conditional Generation with a Question-Answering Blueprint (2023.tacl-1)

Challenge:	a new study examines the current state of knowledge about the BERT model . the model is a stack of transformer encoder layers that are based on multiple self-attention ''heads''
Approach:	They present a survey of over 150 studies of the popular Transformer-based model BERT . they discuss the current state of knowledge about how BERT works and how it is represented .
Outcome:	The proposed model is based on the Transformer-based model with state-of-the-art results . the proposed model has little cognitive motivation and is too small to perform ablation studies .

Copied to clipboard

Shashi Narayan, Joshua Maynez, Reinald Kim Amplayo, Kuzman Ganchev, Annie Louis, Fantine Huot, Anders Sandholm, Dipanjan Das, Mirella Lapata

Question Answering

Challenge:	Neural generation models often struggle to identify which content units are salient.
Approach:	They propose a new conceptualization of text plans as a sequence of question-answer pairs . they propose QA blueprints as QA proxy for content selection and planning .
Outcome:	The proposed model improves existing datasets with QA blueprints as proxy for content selection and planning.

Being Right for Whose Right Reasons? (2023.acl-long)

Copied to clipboard

Terne Sasha Thorn Jakobsen, Laura Cabello, Anders Søgaard

Parameter-Efficient Tuning with Special Token Adaptation (2023.eacl-main)

Challenge:	Existing work has failed to acknowledge that what counts as a rationale is subjective.
Approach:	They propose to use demographic annotations to augment existing datasets to ask what demographics our models align with and whose reasoning patterns they align with.
Outcome:	The proposed model rationales align better with older and/or white annotators, and are biased towards older and white anorators.

Copied to clipboard

Xiaocong Yang, James Y. Huang, Wenxuan Zhou, Muhao Chen

Compressing Large-Scale Transformer-Based Models: A Case Study on BERT (2021.tacl-1)

Challenge:	a recent study shows that parameter-efficient tuning is a challenge for multitask deployments.
Approach:	They propose a parameter-efficient tuning technique that only updates a small subset of parameters when adapting a pretrained model to downstream tasks.
Outcome:	The proposed method achieves comparable performance to fine-tuning in natural language understanding tasks including text classification and NER with only 0.029% of parameters trained.

Copied to clipboard

Prakhar Ganesh, Yao Chen, Xin Lou, Mohammad Ali Khan, Yin Yang, Hassan Sajjad, Preslav Nakov, Deming Chen, Marianne Winslett

Alignment before Awareness: Towards Visual Question Localized-Answering in Robotic Surgery via Optimal Transport and Answer Semantics (2024.lrec-main)

Challenge:	Popular pre-trained Transformers have improved performance for various NLP tasks by sizable margins, but are too resource-hungry and computation-intensive to suit low-capacity devices or applications with strict latency requirements.
Approach:	They present a literature review of the compression of Transformers, focusing on the popular BERT model, which has attracted considerable research attention.
Outcome:	The proposed models improve Sentiment analysis, paraphrase detection, machine reading comprehension, question answering, text summarization, and other tasks by sizable margins.

Copied to clipboard

Zhihong Zhu, Yunyan Zhang, Xuxin Cheng, Zhiqi Huang, Derong Xu, Xian Wu, Yefeng Zheng

When a sentence does not introduce a discourse entity, Transformer-based models still sometimes refer to it (2022.naacl-main)

Challenge:	Recent models for visual question localized-answering (VQLA) lack the ability to relate these answers to their localization at an instance level.
Approach:	They propose a model which introduces optimal transport to achieve bidirectional and fine-grained alignment between images and questions, enabling more precise localization.
Outcome:	The proposed model outperforms state-of-the-art models on two widely-used datasets on surgical scenes and surgical instruments.

Copied to clipboard

Sebastian Schuster, Tal Linzen

What Clued the AI Doctor In? On the Influence of Data Source and Quality for Transformer-Based Medical Self-Disclosure Detection (2023.eacl-main)

Challenge:	Existing theories on how humans track discourse entities are based on the idea that humans maintain explicit memory representations for each entity that encode all properties of an entity and its relation to other entities.
Approach:	They adapt the psycholinguistic assessment of language models paradigm to higher-level linguistic phenomena and introduce an English evaluation suite that targets the knowledge of the interactions between sentential operators and indefinite NPs.
Outcome:	The evaluation suite targets the knowledge of the interactions between sentential operators and indefinite NPs and the models are challenged by multiple NP's and their behavior is not systematic.

Copied to clipboard

Mina Valizadeh, Xing Qian, Pardis Ranjbar-Noiey, Cornelia Caragea, Natalie Parde

DecBERT: Enhancing the Language Understanding of BERT with Causal Attention Masks (2022.findings-naacl)

Challenge:	Recognizing medical self-disclosure is important in many healthcare contexts, but it has been under-explored by the NLP community.
Approach:	They analyze a social media-based task to expand existing medical self-disclosure corpus and compare Transformer-based models to determine their merits.
Outcome:	The proposed dataset outperforms the state-of-the-art dataset for this task by 16.73%.

Copied to clipboard

Ziyang Luo, Yadong Xi, Jing Ma, Zhiwei Yang, Xiaoxi Mao, Changjie Fan, Rongsheng Zhang

Better Language Model with Hypernym Class Prediction (2022.acl-long)

Challenge:	Experimental results show that Transformer Encoder model can't automatically capture word order, so explicit position embeddings are required to be fed into the target model.
Approach:	They propose a Transformer-based language model DecBERT that uses a causal attention mask to capture word order.
Outcome:	The proposed model improves on the GLUE language understanding benchmark and accelerates the pre-training process.

Copied to clipboard

He Bai, Tong Wang, Alessandro Sordoni, Peng Shi

NLQuAD: A Non-Factoid Long Question Answering Data Set (2021.eacl-main)

Challenge:	Class-based language models (LMs) have been devised to address context sparsity in n-gram LMs for decades.
Approach:	They propose to use class-based prediction to improve generalization for rare words by annealing from predicting the class to token prediction during training.
Outcome:	The proposed model improves perplexity without sacrificing performance on rare words.

Copied to clipboard

Amir Soleimani, Christof Monz, Marcel Worring

Question Answering

Challenge:	Existing data sets for document-level question answering are limited in their ability to detect short text and require multiple-sentence descriptive answers and opinions.
Approach:	They introduce a new data set with baseline methods for non-factoid long question answering . they compare BERT, RoBERTa, and Longformer models to establish baseline performances .
Outcome:	Experimental results show that Longformer outperforms the other architectures but human evaluations show that it is far behind the human upper bound.

ArabicTransformer: Efficient Large Arabic Language Model with Funnel Transformer and ELECTRA Objective (2021.findings-emnlp)

Copied to clipboard

Sultan Alrowili, Vijay Shanker

KLEJ: Comprehensive Benchmark for Polish Language Understanding (2020.acl-main)

Challenge:	Existing solutions to reduce the cost of pretraining Transformer-based models are expensive especially for large-scale models.
Approach:	They propose to reduce the cost of pre-training Transformer-based models by compressing the sequence of hidden states inside Transformer architecture.
Outcome:	The proposed model achieves state-of-the-art on several Arabic downstream tasks despite using less computational resources compared to other BERT-based models.

Copied to clipboard

Piotr Rybak, Robert Mroczkowski, Janusz Tracz, Ireneusz Gawlik

DYLE: Dynamic Latent Extraction for Abstractive Long-Input Summarization (2022.acl-long)

Challenge:	Recent introduction of robust, general-purpose models for fine-tuning has enabled improvements in general natural language understanding (NLU) but such benchmarks are only available for a handful of languages.
Approach:	They propose a multi-task benchmark for the Polish language understanding with an online leaderboard . they also propose GLUE, a task for named entity recognition and sentiment analysis .
Outcome:	The proposed model performs best on three out of nine tasks in the Polish language . the proposed model is also used in an e-commerce domain to analyze the sentiments of users .

Copied to clipboard

Ziming Mao, Chen Henry Wu, Ansong Ni, Yusen Zhang, Rui Zhang, Tao Yu, Budhaditya Deb, Chenguang Zhu, Ahmed Awadallah, Dragomir Radev

Speechformer: Reducing Information Loss in Direct Speech Translation (2021.emnlp-main)

Challenge:	Existing models struggle with summarizing long text due to high memory complexity of the full self-attention.
Approach:	They propose a dynamic latent extraction approach for abstractive long-input summarization that treats extracted text snippets as latent variables and allows dynamic attention weights during decoding.
Outcome:	The proposed method outperforms existing methods on GovReport, QMSum, and arXiv while yielding strong results on arX.

Copied to clipboard

Sara Papi, Marco Gaido, Matteo Negri, Marco Turchi

Speech & Audio in NLP

Challenge:	Current approaches to speech-to-text translation (ST) use a pipeline of two sub-components - an automatic speech recognition (ASR) and a machine translation (MT) model.
Approach:	They propose an architecture that avoids initial lossy compression and aggregates information only at a higher level according to more informed linguistic criteria.
Outcome:	The proposed architecture achieves gains of up to 0.8 BLEU on the standard MuST-C corpus and up to 4.0 BLUE in a low resource scenario.

Applications of BERT Models Towards Automation of Clinical Coding in Icelandic (2024.findings-naacl)

Copied to clipboard

Haraldur Orri Hauksson, Hafsteinn Einarsson

ViGoEmotions: A Benchmark Dataset For Fine-grained Emotion Detection on Vietnamese Texts (2026.eacl-long)

Challenge:	Traditionally, clinical coding is manual and laborintensive task prone to human error.
Approach:	They analyze 25 years of electronic health records from the Landspitali University Hospital in Icelandic to explore the potential of using NLP for clinical coding.
Outcome:	The best-performing model achieves competitive results in micro and macro F1 scores, with label attention contributing significantly to its success.

Copied to clipboard

Tran Quang Hung, Pham Tien Nam, Son T. Luu, Kiet Van Nguyen

Towards Understanding Counseling Conversations: Domain Knowledge and Large Language Models (2024.findings-eacl)

Challenge:	Recent advances in NLP have greatly improved outcomes in emotion prediction and harmful content detection.
Approach:	They propose to classify Vietnamese comments into 27 distinct emotions using a model-based lexical normalization system and a transformer-based model.
Outcome:	The proposed corpus of 20,664 social media comments is based on a novel model that can support multiple architectures, but its quality and preprocessing strategies remain key factors influencing performance.

Copied to clipboard

Younghun Lee, Dan Goldwasser, Laura Schwab Reese

Better Explain Transformers by Illuminating Important Information (2024.findings-eacl)

Challenge:	Existing language models such as Transformer-based models fail to predict the conversation outcome.
Approach:	They propose to integrate human-annotated domain knowledge and LLM-generated features to provide richer context to counseling conversations.
Outcome:	The proposed model improves by 15% when combined with human-annotated domain knowledge and LLM-generated features.

Copied to clipboard

Linxin Song, Yan Cui, Ao Luo, Freddy Lecue, Irene Li

Explainability & Interpretability in NLP

Challenge:	Existing explanations focus on the input and output of the Transformers, resulting in confusing results.
Approach:	They propose to highlight important information and eliminate irrelevant information by a refined information flow on top of the layer-wise relevance propagation method.
Outcome:	The proposed method outperforms baseline models on classification and question-answering datasets with over 3% to 33% improvement on explanation metrics.

Hero-Gang Neural Model For Named Entity Recognition (2022.naacl-main)

Copied to clipboard

Jinpeng Hu, Yaling Shen, Yang Liu, Xiang Wan, Tsung-Hui Chang

DecompX: Explaining Transformers Decisions by Propagating Token Decomposition (2023.acl-long)

Challenge:	Named entity recognition (NER) is a fundamental and important task in natural language processing.
Approach:	They propose a novel Hero-Gang Neural structure to leverage both global and local information to promote NER by using a Transformer-based encoder and a Gang module.
Outcome:	The proposed model can extract local features and position information from the Hero and Gang modules, and it performs on multiple datasets.

Copied to clipboard

Ali Modarressi, Mohsen Fayyaz, Ehsan Aghazadeh, Yadollah Yaghoobzadeh, Mohammad Taher Pilehvar

Explainability & Interpretability in NLP

Challenge:	Existing vector-based explanation methods for Transformer-based models are limited in their ability to explain the decisions of multiple layers.
Approach:	They propose a vector-based explanation method based on the construction of decomposed token representations and their successive propagation throughout the model without mixing them in between layers.
Outcome:	The proposed method outperforms existing vector-based and gradient-based methods on transformer-based models by a wide margin.

Contributions of Transformer Attention Heads in Multi- and Cross-lingual Tasks (2021.acl-long)

Copied to clipboard

Weicheng Ma, Kai Zhang, Renze Lou, Lili Wang, Soroush Vosoughi

fBERT: A Neural Transformer for Identifying Offensive Content (2021.findings-emnlp)

Challenge:	Prior research has found that only a few attention heads are important in each mono-lingual NLP task and pruning the remaining heads leads to comparable or improved performance of the model.
Approach:	They examine the relative importance of attention heads in Transformer-based models to aid their interpretability in cross-lingual and multi-lingual tasks.
Outcome:	The proposed model performs better with the remaining heads pruned than with the other models, the authors show .

Copied to clipboard

Diptanu Sarkar, Marcos Zampieri, Tharindu Ranasinghe, Alexander Ororbia

How Certain is Your Transformer? (2021.eacl-main)

Challenge:	Existing models such as BERT, XLNET, and XLM-R have outperformed other neural architectures and statistical learning methods in the identification of offensive language and hate speech.
Approach:	They present a BERT model retrained on SOLID, the largest English offensive language identification corpus available with over 1.4 million offensive instances.
Outcome:	The proposed model outperforms models trained on SOLID, the largest English offensive language identification corpus available with over 1.4 million offensive instances.

Copied to clipboard

Artem Shelmanov, Evgenii Tsymbalov, Dmitri Puzyrev, Kirill Fedyanin, Alexander Panchenko, Maxim Panov

Challenging the Transformer-based models with a Classical Arabic dataset: Quran and Hadith (2022.lrec-1)

Challenge:	Obtaining reliable uncertainty estimations for such neural networks (NNs) is challenging due to the huge number of parameters in these deep learning models.
Approach:	They propose to use Monte Carlo dropout to estimate uncertainty for Transformer-based models and construct inexpensive estimates using Determinantal Point Processes.
Outcome:	The proposed estimates improve the quality of detection of error-prone instances.

Copied to clipboard

Shatha Altammami, Eric Atwell

Unsupervised Natural Language Inference Using PHL Triplet Generation (2022.findings-acl)

Challenge:	Existing benchmark datasets have a low readability index which does not reflect real-world complex data.
Approach:	They constructed a dataset of Quran-verse and Hadith-teaching pairs by consulting sources of reputable religious experts.
Outcome:	The proposed models performed on a binary classification task to identify whether two pieces of CA text convey the same underlying message.

Copied to clipboard

Neeraj Varshney, Pratyay Banerjee, Tejas Gokhale, Chitta Baral

Automatic Annotation of Grammaticality in Child-Caregiver Conversations (2024.lrec-main)

Challenge:	In some cases, training samples may not be available or collecting them could be time-consuming and resource-intensive.
Approach:	They propose a procedural approach that leverages sentence transformations to collect PHL triplets for training NLI models.
Outcome:	The proposed model outperforms existing models on several NLI benchmarks with a set of sentence transformations.

Copied to clipboard

Mitja Nikolaus, Abhishek Agrawal, Petros Kaklamanis, Alex Warstadt, Abdellah Fourtassi

Dialogue Systems & Conversational Agents

Challenge:	Existing methods for analyzing child language acquisition have been tedious and inconsistent.
Approach:	They propose a coding scheme for context-dependent grammaticality in child-caregiver conversations and annotate 4,000 utterances from a large corpus of transcribed conversations.
Outcome:	The proposed method achieves human inter-annotation agreement levels and is faster and reproducible than manual methods.

Once is Enough: A Light-Weight Cross-Attention for Fast Sentence Pair Modeling (2023.emnlp-main)

Copied to clipboard

Yuanhang Yang, Shiyi Qi, Chuanyi Liu, Qifan Wang, Cuiyun Gao, Zenglin Xu

Challenge:	Recent studies suggest that transformer-based models perform cross-attention over input pairs, leading to computational cost.
Approach:	They propose a lightweight cross-attention mechanism that performs query encoding only once while modeling the query-candidate interaction in parallel.
Outcome:	The proposed model speeds up sentence pairing by over 113x while achieving comparable performance as the more expensive models.

The GINCO Training Dataset for Web Genre Identification of Documents Out in the Wild (2022.lrec-1)

Copied to clipboard

Taja Kuzman, Peter Rupnik, Nikola Ljubešić

MAFiD: Moving Average Equipped Fusion-in-Decoder for Question Answering over Tabular and Textual Data (2023.findings-eacl)

Challenge:	GINCO is a new training dataset for automatic genre identification based on 1,125 crawled Slovenian web documents that consist of 650,000 words.
Approach:	They propose to use 1,125 crawled Slovenian web documents to train a new genre classification system based on a GINCO training dataset .
Outcome:	The proposed classifiers perform better on the 1,125 crawled Slovenian web documents than the existing models and achieve higher scores on the task.

Copied to clipboard

Sung-Min Lee, Eunhwan Park, Daeryong Seo, Donghyeon Jeon, Inho Kang, Seung-Hoon Na

Transformers with Learnable Activation Functions (2023.findings-eacl)

Challenge:	Experimental results show that Transformer-based questions have a "long" hybrid sequence over tabular and textual elements, causing long-range reasoning problems.
Approach:	They propose a moving average-equipped fusion-in-decoder to handle long-range reasoning problems . they use FiD and EMA to combine different levels of reasoning .
Outcome:	Experimental results show that the proposed model increases exact matching and F1 by 1.1 and 1.7 on the blind test set.

Copied to clipboard

Haishuo Fang, Ji-Ung Lee, Nafise Sadat Moosavi, Iryna Gurevych

Geo-BERT Pre-training Model for Query Rewriting in POI Search (2021.findings-emnlp)

Challenge:	Activation functions can reduce the topological complexity of input data and improve model performance.
Approach:	They propose to consider data as a topology with its own shape to simplify its complexity and make it linearly separable in the output space.
Outcome:	The RAF-based Transformer model outperforms its FAF-based counterpart on the GLUE benchmark by 5.71 points and 2.05 points on SQuAD with all available data.

Copied to clipboard

Xiao Liu, Juan Hu, Qi Shen, Huan Chen

Stress Test Evaluation of Transformer-based Models in Natural Language Understanding Tasks (2020.lrec-1)

Challenge:	Existing methods to solve the word mismatch between queries and documents are often inadequate to integrate geographic information into the pre-training model.
Approach:	They propose to train a pre-training model to integrate semantics and geographic information in the pre-trained representations of POIs.
Outcome:	The proposed model achieves excellent accuracy on a wide range of real-world datasets of map services.

Copied to clipboard

Carlos Aspillaga, Andrés Carvallo, Vladimir Araujo

Principled Understanding of Generalization for Generative Transformer Models in Arithmetic Reasoning Tasks (2025.acl-long)

Challenge:	Existing models are weak and take advantage of failures and errors in datasets to improve performance.
Approach:	They evaluate three Transformer-based models in Natural Language Inference and Question Answering tasks to see if they are more robust or have the same flaws as their predecessors.
Outcome:	The proposed models outperform recurrent neural network models to stress tests on both NLI and QA tasks.

Copied to clipboard

Xingcheng Xu, Zibo Zhao, Haipeng Zhang, Yanqing Yang

Investigating Failures of Automatic Translation in the Case of Unambiguous Gender (2022.acl-long)

Challenge:	Existing models excel in arithmetic reasoning but their generalization capabilities are incompletely understood.
Approach:	They propose a theoretical framework for understanding the generalization behaviors of transformers in arithmetic tasks, focusing on length generalization.
Outcome:	The proposed framework can predict generalization behaviors in transformers with a high translation invariance and base mismatch in modular operations.

Copied to clipboard

Adithya Renduchintala, Adina Williams

Machine Translation

Challenge:	Existing models are unable to make basic deductions regarding how to correctly inflect nouns with grammatical gender.
Approach:	They propose to evaluate NMT models' ability to translate gender morphology correctly in unambiguous contexts across syntactically diverse sentences.
Outcome:	The proposed model was unable to translate gender morphology correctly in unambiguous contexts across syntactically diverse sentences.

Quantifying Context Mixing in Transformers (2023.eacl-main)

Copied to clipboard

Hosein Mohebbi, Willem Zuidema, Grzegorz Chrupała, Afra Alishahi

What Did You Learn To Hate? A Topic-Oriented Analysis of Generalization in Hate Speech Detection (2023.eacl-main)

Challenge:	Self-attention weights and their transformed variants have been used for analyzing token-to-token interactions in Transformer-based models, but they are not faithful to the models’ decisions as they are only one part of an encoder block.
Approach:	They propose a new context mixing score customized for Transformers that provides us with a deeper understanding of how information is mixed at each encoder layer.
Outcome:	The proposed score outperforms other methods in linguistically informed rationales, probing, and faithfulness analysis.

Copied to clipboard

Tom Bourgeade, Patricia Chiril, Farah Benamara, Véronique Moriceau

When BERT Plays the Lottery, All Tickets Are Winning (2020.emnlp-main)

Challenge:	Hate speech detection datasets often use different annotation guidelines, resulting in inconsistencies . authors propose a topic-oriented approach to study generalization across popular hate speech datasets .
Approach:	They propose a topic-oriented approach to study generalization across popular hate speech datasets . they compare Transformer-based models in capturing topic-generic and topic-specific knowledge .
Outcome:	The proposed approach improves the reliability of hate speech detection on social media platforms.

Copied to clipboard

Sai Prasanna, Anna Rogers, Anna Rumshisky

On the weak link between importance and prunability of attention heads (2020.emnlp-main)

Challenge:	Large Transformer-based models are reduced to a smaller number of self-attention heads and layers.
Approach:	They propose to prune BERT self-attention heads and layers to find subnetworks with comparable performance . they also extend this technique to multi-layer perceptrons to find out if they are unstable .
Outcome:	The proposed models are able to achieve 90% of full model performance with structured pruning and similar-sized subnetworks sampled from the rest of the model perform worse.

Copied to clipboard

Aakriti Budhraja, Madhura Pande, Preksha Nema, Pratyush Kumar, Mitesh M. Khapra

Token Dropping for Efficient BERT Pretraining (2022.acl-long)

Challenge:	a large fraction of attention heads can be randomly pruned with limited effect on accuracy, a new study finds . a second study finds no advantage in pruning attention heads identified to be important based on the location of a head .
Approach:	They examine the importance of pruning attention heads on a Transformer-based model . they find no advantage in pruning attention head positions on the BERT model based on location .
Outcome:	The results show that pruning strategies on Transformer and BERT models are not important based on location . the results suggest that interpretation of attention heads does not strongly inform pruning strategies.

Copied to clipboard

Le Hou, Richard Yuanzhe Pang, Tianyi Zhou, Yuexin Wu, Xinying Song, Xiaodan Song, Denny Zhou

Detecting Unassimilated Borrowings in Spanish: An Annotated Corpus and Approaches to Modeling (2022.acl-long)

Challenge:	Existing methods to accelerate pretraining of transformer-based models are computationally expensive and degrade performance on downstream tasks.
Approach:	They propose a "token dropping" method to accelerate the pretraining of transformer-based models by 25% . they leverage the already built-in masked language modeling loss to identify unimportant tokens with practically no computational overhead.
Outcome:	The proposed method reduces the pretraining cost of BERT models by 25% while achieving similar overall performance on downstream tasks.

Copied to clipboard

Elena Álvarez-Mellado, Constantine Lignos

STANKER: Stacking Network based on Level-grained Attention-masked BERT for Rumor Detection on Social Media (2021.emnlp-main)

Challenge:	a corpus of Spanish newswire rich in unassimilated lexical borrowings is used to identify the language of a word.
Approach:	They propose to annotate a corpus of Spanish newswire rich in unassimilated lexical borrowings and evaluate how models perform on this task.
Outcome:	The proposed model outperforms models fed with subword embeddings and Transformer-based embeddables on the Spanish newswire corpus.

Copied to clipboard

Dongning Rao, Xin Miao, Zhihua Jiang, Ran Li

MoLA: MoE LoRA with Layer-wise Expert Allocation (2025.findings-naacl)

Challenge:	Existing models for text classification are limited in performance, resulting in poor rumor detection.
Approach:	They propose to use Chinese microblogs to detect rumors using pre-trained language models and auxiliary features such as comments to mask co-attention.
Outcome:	The proposed model outperforms the state-of-the-art on Weibo20 and three existing social media datasets.

Copied to clipboard

Chongyang Gao, Kezhen Chen, Jinmeng Rao, Ruibo Liu, Baochen Sun, Yawen Zhang, Daiyi Peng, Xiaoyuan Guo, Vs Subrahmanian

Green & Sustainable NLP

Challenge:	Recent efforts to integrate low-rank adaptation (LoRA) with the Mixture-of-Experts (MoE) have achieved performance comparable to full-parameter fine-tuning by tuning much fewer parameters.
Approach:	They propose a parameter-efficient MoE method for low-rank adaptation with the Mixture-of-Experts (MoE) they use layers of LoRA experts to allocate more LoRA expert to middle layers .
Outcome:	The proposed method outperforms baseline models on six well-known NLP and commonsense QA benchmarks on LLAMA-2, Mistral, and Gemma.

Classifying Social Media Users before and after Depression Diagnosis via Their Language Usage: A Dataset and Study (2024.lrec-main)

Copied to clipboard

Falwah Alhamed, Julia Ive, Lucia Specia

H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequences (2021.acl-long)

Challenge:	Mental illness can negatively impact individuals’ quality of life as it is considered one of the causes of years lived with disability and it is related to high suicide rates.
Approach:	They collect first dataset of textual posts by same users before and after being diagnosed with depression and build multiple predictive models based on Transformers and BERT.
Outcome:	The proposed model can be used to detect depression and suicidal thoughts in users who are not diagnosed with depression or suicide.

Copied to clipboard

Zhenhai Zhu, Radu Soricut

Leveraging Unimodal Self-Supervised Learning for Multimodal Audio-Visual Speech Recognition (2022.acl-long)

Challenge:	Existing hierarchical methods to compute attention are superior to sub-quadratic ones . a particular type of attention, called multi-head scaled dot-product attention, is one of the main components of the Transformer architecture .
Approach:	They propose a hierarchical method to compute attention in the Transformer architecture . they perform extensive experiments to show that it captures hierarchic structure in sequences .
Outcome:	The proposed method outperforms sub-quadratic models on the Long Range Arena benchmark by over +6 points on average.

Copied to clipboard

Xichen Pan, Peiyu Chen, Yichen Gong, Helong Zhou, Xinbing Wang, Zhouhan Lin

COMICORDA: Dialogue Act Recognition in Comic Books (2024.lrec-main)

Challenge:	Existing methods for audio-visual speech recognition use extra data to increase performance . a recent study shows that the use of unimodal self-supervised learning improves performance on multimodal tasks.
Approach:	They propose to use unimodal self-supervised learning to train AVSR models on unlabelled unilateral data.
Outcome:	The proposed model improves on lip reading sentences 2 by 30% even without an external language model.

Copied to clipboard

Jiri Martinek, Pavel Kral, Ladislav Lenc, Josef Baloun

SPE Attention: Making Attention Equivariant to Semantic-Preserving Permutation for Code Processing (2025.emnlp-main)

Challenge:	Existing work on dialogue act recognition from images is limited to speech balloon segmentation and optical character recognition.
Approach:	They propose a novel DA recognition approach for comic books using speech balloon segmentation, optical character recognition and DA classification.
Outcome:	The proposed method achieves 98% average precision for speech balloon segmentation and exceeds 70% accuracy for the DA recognition task.

Copied to clipboard

Chengyu Jiao, Shuhao Chen, Yu Zhang

Transformer-based Lexically Constrained Headline Generation (2021.emnlp-main)

Challenge:	Existing approaches to train code processing models to capture symmetry of code . semantic-preserving permutations are not found in natural languages .
Approach:	They propose a mechanism that captures a unique symmetry of code, called the SPE attention . they propose symmetry graphs that are then combined to create a symmetry mask .
Outcome:	The proposed model can be used to analyze code summarization and error detection tasks.

Copied to clipboard

Kosuke Yamada, Yuta Hitomi, Hideaki Tamori, Ryohei Sasano, Naoaki Okazaki, Kentaro Inui, Koichi Takeda

SlimFit: Memory-Efficient Fine-Tuning of Transformer-based Models Using Training Dynamics (2024.naacl-long)

Challenge:	Existing automatic headline generation methods cannot include a given phrase in the generated headline.
Approach:	They propose a Transformer-based method that guarantees to include a given phrase in a generated headline.
Outcome:	The proposed method achieves ROUGE scores comparable to previous methods with Japanese news corpus.

Copied to clipboard

Arash Ardakani, Altan Haan, Shangyin Tan, Doru Thom Popovici, Alvin Cheung, Costin Iancu, Koushik Sen

Mining Complex Patterns of Argumentative Reasoning in Natural Language Dialogue (2025.acl-long)

Challenge:	SlimFit reduces the memory requirements of transformer-based models by analyzing their training dynamics and freezing less-contributory layers during fine-tuning.
Approach:	They propose a tool that analyzes transformer-based models and freezes less-contributory layers during fine-tuning to reduce the overall on-device memory usage.
Outcome:	SlimFit reduces the memory requirements of transformer-based models by analyzing their training dynamics and freezing less-contributory layers during fine-tuning.

Copied to clipboard

Ramon Ruiz-Dolz, Zlata Kikteva, John Lawrence

Dialogue Systems & Conversational Agents

Challenge:	Argumentation scheme mining is the task of automatically identifying reasoning mechanisms behind argument inferences.
Approach:	They propose to create a corpus of 441 arguments annotated with 24 argumentation schemes and leverage the capabilities of LLMs and Transformer-based models to validate their applicability in real-world scenarios.
Outcome:	The proposed corpus of arguments is pre-trained on a large corpus containing textbook-like argumentation schemes and validates their applicability in real-world scenarios.

TinyBERT: Distilling BERT for Natural Language Understanding (2020.findings-emnlp)

Copied to clipboard

Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, Qun Liu

Incorporating Residual and Normalization Layers into Analysis of Masked Language Models (2021.emnlp-main)

Challenge:	Pre-trained language models are computationally expensive and difficult to efficiently execute on resource-restricted devices.
Approach:	They propose a Transformer distillation method that performs Transformer distillations at pre-training and task-specific learning stages.
Outcome:	The proposed method accelerates inference and reduces model size while maintaining accuracy.

Copied to clipboard

Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, Kentaro Inui

Sketching as a Tool for Understanding and Accelerating Self-attention for Long Sequences (2022.naacl-main)

Challenge:	Transformer architecture is composed of multi-head attention, which has been extensively analyzed.
Approach:	They extended the scope of the analysis of Transformers from solely the attention patterns to the whole attention block, i.e., multi-head attention, residual connection, and layer normalization.
Outcome:	The proposed method incorporates the whole attention block, i.e., multi-head attention, residual connection, and layer normalization into the analysis.

Copied to clipboard

Yifan Chen, Qi Zeng, Dilek Hakkani-Tur, Di Jin, Heng Ji, Yun Yang

Czech Dataset for Complex Aspect-Based Sentiment Analysis Tasks (2024.lrec-main)

Challenge:	Existing models for long sequences are not efficient due to the quadratic space and time complexity of the self-attention modules.
Approach:	They propose to reduce the quadratic complexity to linear (modulo logarithmic factors) by low-dimensional projection and row selection.
Outcome:	The proposed methods outperform transformer-based models with smaller time/space footprint on the Long Range Arena benchmark.

Copied to clipboard

Jakub Šmíd, Pavel Přibáň, Ondrej Prazak, Pavel Kral

Decoding Probing: Revealing Internal Linguistic Structures in Neural Language Models Using Minimal Pairs (2024.lrec-main)

Challenge:	3.1K reviews are manually annotated for aspect-based sentiment analysis (ABSA) ABSA is a fine-grained task that aims to identify the sentiment associated with each aspect or characteristic of a text.
Approach:	They propose a new Czech dataset for aspect-based sentiment analysis . the new dataset is built upon the older Czech dataset . authors provide 24M reviews without annotations suitable for unsupervised learning .
Outcome:	The proposed dataset is built upon the older dataset, but is specifically designed for more complex tasks.

Copied to clipboard

Linyang He, Peili Chen, Ercong Nie, Yuanning Li, Jonathan R. Brennan

Efficient Classification of Long Documents via State-Space Models (2023.emnlp-main)

Challenge:	a new method is being developed to probe internal linguistic characteristics in neural language models layer by layer .
Approach:	They propose a method that uses minimal pairs benchmark to probe internal linguistic characteristics in neural language models layer by layer.
Outcome:	The proposed method captures grammaticality labels in language models layer by layer . it is based on the cognitive neurosciences of the brain and its representations as "neural activations".

Copied to clipboard

Peng Lu, Suyuchen Wang, Mehdi Rezagholizadeh, Bang Liu, Ivan Kobyzev

Effective Pretraining Objectives for Transformer-based Autoencoders (2022.findings-emnlp)

Challenge:	Existing transformer-based models can only process long documents with limited computational resources due to their quadratic computation time and space.
Approach:	They propose to use state-space models for long document classification tasks instead of using sparse or hierarchical structures to solve this problem.
Outcome:	The proposed model performs comparable to self-attention models while being 36% more efficient.

Copied to clipboard

Luca Di Liello, Matteo Gabburo, Alessandro Moschitti

What do Toothbrushes do in the Kitchen? How Transformers Think our World is Structured (2022.naacl-main)

Challenge:	ELECTRA is more accurate than BERT, but it is not clear if this is due to its innovative architecture or to the long and extensive training, which highly increases the computation cost for obtaining the final language model.
Approach:	They propose to replace BERT’s Masked Language Modeling objective (MLM) with Token Detection (TD) by using a statistical approach to generate light tokens.
Outcome:	The proposed method can replace ELECTRA's computationally heavy generators without a significant drop in performance.

Copied to clipboard

Alexander Henlein, Alexander Mehler

DimA: A Parameter-efficient Fine-tuning Method with Knowledge Transfer Based on Transformer (2024.lrec-main)

Challenge:	Recent research reveals that transformer-based models are biased towards extracting knowledge about object relations.
Approach:	They propose to use transformer-based models to extract knowledge about object relations to investigate whether they can be used to extract object relations.
Outcome:	The proposed models outperform static models in many respects and perform much worse than similarity measures and classifiers.

Copied to clipboard

Wenxuan Zhang, Min Huang, Zhuoyang Song, Qinghai Miao

Highlight-Transformer: Leveraging Key Phrase Aware Attention to Improve Abstractive Multi-Document Summarization (2021.findings-acl)

Challenge:	Pre-trained language models (PLMs) have demonstrated impressive performance across various downstream tasks, but fine-tuning is computationally expensive and storage-intensive.
Approach:	They propose a parameter-efficient method called DimA which enhances the transformer architecture by increasing the dimensionality.
Outcome:	The proposed method achieves state-of-the-art results in GLUE and XSUM tasks while utilizing less than 1% of the original model’s parameters.

Copied to clipboard

Shuaiqi Liu, Jiannong Cao, Ruosong Yang, Zhiyuan Wen

Transformers in the loop: Polarity in neural models of language (2022.acl-long)

Challenge:	Existing models do not consider key phrases in determining attention weights of self-attention . Existing work does not consider the importance of key phrases when determining weights .
Approach:	They propose a model with highlighting mechanism to assign greater attention weights to key phrases . they propose two structures of highlighting attention for each head and the multihead highlighting . experimental results show that their proposed model significantly outperforms the baseline model .
Outcome:	The proposed model outperforms the baseline models on a multi-news dataset.

Copied to clipboard

Lisa Bylinina, Alexey Tikhonov

GradTS: A Gradient-Based Automatic Auxiliary Task Selection Method Based on Transformer Networks (2021.emnlp-main)

Challenge:	Recent Transformer-based language representation models (LRMs) show impressive results on practical text analysis tasks, but do they have access to complex linguistic notions?
Approach:	They propose to use polarity as a case study to compare metrics derived from language models to human judgments obtained in psycholinguistic experiments.
Outcome:	The proposed model is more accurate than linguistic theory predictions for polarity, and allows us to use language models to discover new insights into natural language grammar beyond existing linguistic theories.

Copied to clipboard

Weicheng Ma, Renze Lou, Kai Zhang, Lili Wang, Soroush Vosoughi

Long-Span Summarization via Local Attention and Content Selection (2021.acl-long)

Challenge:	A key problem in multi-task learning (MTL) research is how to select high-quality auxiliary tasks automatically.
Approach:	They propose an automatic auxiliary task selection method based on gradient calculation in Transformer-based models that improves MT-DNN performance.
Outcome:	The proposed method improves MT-DNN performance on 8 natural language understanding (GLUE) tasks, while costing less than AUTOSEM and comparable GPU consumption.

Copied to clipboard

Potsawee Manakul, Mark Gales

An Exploration of Mamba for Speech Self-Supervised Models (2026.acl-long)

Challenge:	Transformer-based models are state-of-the-art for a wide range of natural language processing tasks, including document summarization.
Approach:	They exploit large pre-trained transformer-based models and address long-span dependencies in abstractive summarization using two methods: local self-attention; and explicit content selection.
Outcome:	The proposed models achieve state-of-the-art on Spotify Podcast, arXiv, and PubMed datasets.

Copied to clipboard

Tzu-Quan Lin, Heng-Cheng Kuo, Tzu-Chieh Wei, Hsi-Chun Cheng, Chun Wei Chen, Hsien-Fu Hsiao, Yu Tsao, Hung-yi Lee

Fine-Grained Controllable Text Generation Using Non-Residual Prompting (2022.acl-long)

Challenge:	Mamba-based SSL models are promising for long-sequence modeling, speech unit extraction, and speech self-supervised learning.
Approach:	They propose to use Mamba-based HuBERT models as an alternative to Transformer-based SSL architectures.
Outcome:	The proposed models outperform Transformer-based models in language modeling tasks while showing superior performance on streaming ASR.

Copied to clipboard

Fredrik Carlsson, Joey Öhman, Fangyu Liu, Severine Verlinden, Joakim Nivre, Magnus Sahlgren

Symmetric Dot-Product Attention for Efficient Training of BERT Language Models (2024.findings-acl)

Challenge:	Existing approaches to control the text generation process are not expressive enough.
Approach:	They propose an encoder-decoder architecture that enables intermediate text prompts at arbitrary time steps.
Outcome:	The proposed architecture is expressive and versatile on multiple experimental settings.

Copied to clipboard

Martin Courtois, Malte Ostendorff, Leonhard Hennig, Georg Rehm

On Robustness of Finetuned Transformer-based NLP Models (2023.findings-emnlp)

Challenge:	Transformer-based models are stretched to enormous sizes, requiring increasingly larger training datasets and unsustainable amount of compute resources.
Approach:	They propose an alternative compatibility function for the Transformer-based attention mechanism that exploits an overlap in the learned representation of the traditional scaled dot-product attention mechanism.
Outcome:	The proposed model achieves 79.36 on the GLUE benchmark against 78.74 for the traditional implementation and reduces the number of trainable parameters by 6%.

Copied to clipboard

Pavan Kalyan Reddy Neerudu, Subba Oota, Mounika Marreddy, Venkateswara Kagita, Manish Gupta

Robustness in NLP

Challenge:	Pretrained Transformer-based language models have been finetuned for a large number of tasks.
Approach:	They characterize changes between pretrained and finetuned models with CKA and STIR metrics.
Outcome:	The proposed models are more robust to perturbations than BERT and T5 on classification tasks and generation tasks.

EFTNAS: Searching for Efficient Language Models in First-Order Weight-Reordered Super-Networks (2024.lrec-main)

Copied to clipboard

Juan Pablo Munoz, Yi Zheng, Nilesh Jain

Revisiting Transformer-based Models for Long Document Classification (2022.findings-emnlp)

Challenge:	Depending on the size of transformer-based models, they can be restricted from deployment in resource-constrained environments.
Approach:	They propose to combine neural architecture search and network pruning techniques to generate and train weight-sharing super-networks that contain efficient transformer-based models.
Outcome:	The proposed model achieves high-performing, high-performance subnetworks on the general language understanding evaluation and the Stanford Question Answering Dataset.

Copied to clipboard

Xiang Dai, Ilias Chalkidis, Sune Darkner, Desmond Elliott

Adaptive Attention for Sparse-based Long-sequence Transformer (2023.findings-acl)

Challenge:	Recent literature in text classification is biased towards short text sequences . multi-page multi-paragraph documents cannot be efficiently encoded by vanilla transformers based on short text.
Approach:	They compare different Transformer-based Long Document Classification approaches to mitigate the computational overhead of vanilla transformers to encode much longer text.
Outcome:	The proposed models can process longer text and provide practical advice for long document classification tasks.

Copied to clipboard

Xuanyu Zhang, Zhepeng Lv, Qing Yang

Space Efficient Context Encoding for Non-Task-Oriented Dialogue Generation with Graph Attention Transformer (2021.acl-long)

Challenge:	Recent studies show that Transformers can process longer sequences because of their complexity and time scales quadratic to the sequence length.
Approach:	They propose an efficient Transformer model with adaptive attention that can select useful tokens automatically in sparse attention by learnable position vectors.
Outcome:	The proposed model can select useful tokens automatically in sparse attention by learnable position vectors.

Copied to clipboard

Fabian Galetzka, Jewgeni Rose, David Schlangen, Jens Lehmann

Investigating the Relationship Between Romanian Financial News and Closing Prices from the Bucharest Stock Exchange (2022.lrec-1)

Challenge:	Recent Transformer-based models aim to integrate fixed background context into non-task-oriented dialogue systems, but the context length is fixed in these architectures, which restricts how much background or dialogue context can be kept.
Approach:	They propose a more concise encoding for background context structured in the form of knowledge graphs by expressing the graph connections through restrictions on the attention weights.
Outcome:	The proposed architecture reduces space requirements without negative effects on the precision of reproduction of knowledge and perceived consistency.

Copied to clipboard

Ioan-Bogdan Iordache, Ana Sabina Uban, Catalin Stoean, Liviu P. Dinu

ResFormer: All-Time Reservoir Memory for Long Sequence Classification (2025.emnlp-main)

Challenge:	a new data set is used to extract information related to one company . a model that is based on previous information about transactions is not enough .
Approach:	They use a Romanian financial news website to extract only information related to one company . they use lexicon-based Vader tool, Financial BERT and Transformer-based models .
Outcome:	The proposed model shows that the extracted sentiment scores correlate with stock closing prices . the proposed model is based on data from a Romanian financial news website .

Copied to clipboard

Hongbo Liu, Jia Xu

Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encoding (2024.findings-emnlp)

Challenge:	Existing models with quadratic time and memory complexity restrict input length . however, analyzing extensive sequential contexts is challenging .
Approach:	They propose a neural network architecture that captures contextual dependencies in linear time and a nonlinear readout to model short-term dependencies within sentences.
Outcome:	The proposed model outperforms baseline models on EmoryNLP datasets and on IEMOCAP and MultiWOZ datasets.

Copied to clipboard

Liang Zhao, Xiachong Feng, Xiaocheng Feng, Weihong Zhong, Dongliang Xu, Qing Yang, Hongtao Liu, Bing Qin, Ting Liu

Quantity doesn’t buy quality syntax with neural language models (D19-1)

Challenge:	Existing methods to enhance length extrapolation of large language models have been developed, but a systematic survey is lacking.
Approach:	They propose to examine the effects of positional encoding on length extrapolation.
Outcome:	The proposed methods improve the extrapolation of large language models, but they are still lacking a systematic survey.

Copied to clipboard

Marten van Schijndel, Aaron Mueller, Tal Linzen

Connecting the Dots: What Graph-Based Text Representations Work Best for Text Classification using Graph Neural Networks? (2023.findings-emnlp)

Challenge:	Recurrent neural network language models can learn to predict upcoming words with remarkably low perplexity . but in syntactically complex contexts, they often assign unexpectedly high probabilities to ungrammatical words .
Approach:	They investigate whether recurrent neural networks can learn to predict upcoming words with remarkably low perplexity.
Outcome:	The proposed models perform worse than GPT and BERT in some constructions than LSTMs in other contexts.

Copied to clipboard

Margarita Bugueño, Gerard de Melo

VisualRWKV: Exploring Recurrent Neural Networks for Visual Language Models (2025.coling-main)

Challenge:	Graph Neural Networks have been used for text classification, but only in domains with limited data characteristics.
Approach:	They compare graph representation methods for text classification using different architectures and setups.
Outcome:	The proposed graph representation methods outperform other models in document comprehension tasks.

Copied to clipboard

Haowen Hou, Peigen Zeng, Fei Ma, Fei Richard Yu

PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation (2020.emnlp-main)

Challenge:	Visual Language Models (VLMs) have been gaining popularity with large language models, but few attempts have been made to incorporate efficient linear Recurrent Neural Networks (RNNs) into VLMs.
Approach:	They propose a linear RNN model with a data-dependent recurrence and sandwich prompts to enhance modeling capabilities and a 2D image scanning mechanism to enrich the processing of visual sequences.
Outcome:	The proposed model achieves competitive performance compared to Transformer-based models on various benchmarks.

Copied to clipboard

Bin Bi, Chenliang Li, Chen Wu, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, Luo Si

Natural Language Interfaces

Challenge:	Existing techniques for natural language understanding and generation use autoencoding and/or autoregressive objectives to train models.
Approach:	They propose a self-supervised pre-training scheme that pre-trains an autoencoding and autoregressive language model on a large unlabeled corpus for generating new text conditioned on context.
Outcome:	The proposed scheme achieves state-of-the-art results on a variety of language generation benchmarks covering generative question answering, abstractive summarization and conversational response generation.

Enriching and Controlling Global Semantics for Text Summarization (2021.emnlp-main)

Copied to clipboard

Thong Nguyen, Anh Tuan Luu, Truc Lu, Tho Quan

Accelerating Toeplitz Neural Network with Constant-time Inference Complexity (2023.emnlp-main)

Challenge:	Abstractive summarization models have been proven effective in creating fluent and informative summaries, but they suffer from the short-range dependency problem, causing them to produce summary that miss the key points of document.
Approach:	They propose a neural topic model empowered with normalizing flow to capture global semantics of the document and integrate them into the summarization model.
Outcome:	The proposed model outperforms state-of-the-art summarization models on five common text summarizing datasets, namely CNN/DailyMail, XSum, Reddit TIFU, arXiv, and PubMed.

Copied to clipboard

Zhen Qin, Yiran Zhong

ChuLo: Chunk-Level Key Information Representation for Long Document Understanding (2025.findings-acl)

Challenge:	Toeplitz Neural Networks outperform commonly used Transformer-based models while benefiting from log-linear space-time complexities.
Approach:	They propose to convert TNNs to SSMs during inference to combine strengths of TNN and SSM approaches.
Outcome:	The proposed method outperforms most Transformer-based models while retaining the advantage of constant inference complexity.

Copied to clipboard

Yan Li, Caren Han, Yue Dai, Feiqi Cao

PoliBERTweet: A Pre-trained Language Model for Analyzing Political Content on Twitter (2022.lrec-1)

Challenge:	Traditional approaches to truncate inputs, sparse self-attention, and chunking often lead to information loss and hinder the model’s ability to capture long-range dependencies.
Approach:	They propose a novel chunk representation method that uses unsupervised keyphrase extraction to group input tokens to retain core document content while reducing input length.
Outcome:	The proposed method minimizes information loss and improves the efficiency of Transformer-based models.

Copied to clipboard

Kornraphop Kawintiranon, Lisa Singh

Analyzing Transformers in Embedding Space (2023.acl-long)

Challenge:	Pre-trained domain-specific models are useful for understanding domain-level contexts.
Approach:	They propose to use a pre-trained language model to better capture domain-specific contexts.
Outcome:	The proposed model outperforms general-purpose models on election-related tasks.

Copied to clipboard

Guy Dar, Mor Geva, Ankit Gupta, Jonathan Berant

On the Expressivity Role of LayerNorm in Transformers’ Attention (2023.findings-acl)

Challenge:	Understanding Transformer-based models has attracted significant attention . a zero-pass approach is feasible for some parameters, and for two-layer attention networks .
Approach:	They propose a theoretical framework where parameters of a trained Transformer are interpreted by projecting them into the embedding space.
Outcome:	The proposed framework shows that pre-trained and fine-tuned models can be interpreted in embedding space.

Copied to clipboard

Shaked Brody, Uri Alon, Eran Yahav

Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification (2023.findings-emnlp)

Challenge:	Layer Normalization (LayerNorm) is an inherent component in all Transformer-based models.
Approach:	They propose to use LayerNorm to normalize the activations during the forward pass and their gradients during the backward pass.
Outcome:	The proposed model is able to express the multi-head attention layer that follows it in a d-1 space and scales to the same norm of d.

Copied to clipboard

Jungmin Yun, Mihyeon Kim, Youngbin Kim

Generalized Attention Flow: Feature Attribution for Transformer Models via Maximum Flow (2025.acl-long)

Challenge:	Pre-trained transformers suffer from a computationally expensive self-attention mechanism that interacts with all tokens, including those unfavorable to classification performance.
Approach:	They propose to integrate token pruning and token combining strategies to improve model performance and reduce computational demands.
Outcome:	Experiments with various datasets show that the proposed model performs better than baseline models, with the best improvement over the existing model.

Copied to clipboard

Behrooz Azarkhalili, Maxwell W. Libbrecht

Interpreting and Exploiting Functional Specialization in Multi-Head Attention under Multi-task Learning (2023.emnlp-main)

Challenge:	Existing approaches to feature attributions rely on attention weights and attention weightings.
Approach:	They propose a feature attribution method that replaces attention weights with the generalized Information Tensor to enhance the performance of Transformer-based models.
Outcome:	The proposed method outperforms state-of-the-art feature attribution methods on sequence classification tasks and provides a more reliable interpretation of Transformer model outputs.

Copied to clipboard

Chong Li, Shaonan Wang, Yunhao Zhang, Jiajun Zhang, Chengqing Zong

PAD: A Robustness Enhancement Ensemble Method via Promoting Attention Diversity (2024.lrec-main)

Challenge:	Experimental results show that multi-head attention module evolves functional specialization after multi-task training.
Approach:	They propose a method to quantify the degree of functional specialization in multi-head attention . they propose 'multi-task training' method to increase functional specialisation and mitigate negative information transfer .
Outcome:	The proposed method increases functional specialization and mitigates negative information transfer in multi-task learning without adding any parameters.

Copied to clipboard

Yuting Yang, Pei Huang, Feifei Ma, Juan Cao, Jintao Li

Robustness in NLP

Challenge:	Existing approaches to enhance robustness of deep neural networks focus on perturbation . weak robustness is a problem for many types of adversarial attacks, authors say .
Approach:	They propose a lightweight framework for enhancing robustness by perturbing parameters of a model and diversifying adversarial example distributions among different models.
Outcome:	The proposed method can improve robustness against adversarial attacks while maintaining accuracy on clean data.

I-GUARD: Interpretability-Guided Parameter Optimization for Adversarial Defense (2025.findings-emnlp)

Copied to clipboard

Mamta Mamta, Oana Cocarascu

IRIS: Interpretable Retrieval-Augmented Classification for Long Interspersed Document Sequences (2025.acl-long)

Challenge:	I-Guard leverages model interpretability to identify influential parameters responsible for adversarial misclassifications.
Approach:	They propose a framework to increase the robustness of transformer-based models against adversarial attacks.
Outcome:	The proposed framework improves model robustness against adversarial attacks.

Copied to clipboard

Fengnan Li, Elliot D. Hill, Jiang Shu, Jiaxin Gao, Matthew M. Engelhard